Design of the polytomous simplest complete Q matrix based on the reachability matrix

doi:10.3724/SP.J.1041.2024.01634

Abstract

Abstract:

The identifiability of cognitive diagnosis models relies heavily on the completeness of the Q matrix. However, existing test designs primarily focus on dichotomously-scored items, neglecting the importance of polytomous cognitive diagnostic test design. Moreover, this limitation poses a significant obstacle to the advancement of cognitive diagnosis. To bridge this gap, this paper aimed to introduce novel designs for the construction of polytomous structured and unstructured simplest complete Q matrices (SSCQM/USCQM). Our proposed approach considered all ideal response patterns (IRPs) of knowledge states (KSs) on the reachability matrix as research objects, with the objective of minimizing the number of columns selected from the reachability matrix. This ensured one-to-one correspondence between the set of KSs and the set of IRPs, thereby enhancing the completeness of the SSCQM. Additionally, we derived a polytomous USCQM by considering the relationship between the SSCQM and the sub-matrix of the corresponding identity matrix while ensuring that each row contains at least one “1”. Interestingly, the construction process revealed that there were more USCQMs than SSCQMs. This innovative approach expanded the possibilities for polytomous cognitive diagnostic test design.

Cite this article

TANG Xiaojuan, PENG Zhixia, QIN Shanshan, DING Shuliang, MAO Mengmeng, LI Yu. (2024). Design of the polytomous simplest complete Q matrix based on the reachability matrix. Acta Psychologica Sinica, 56(11), 1634-1650.

share this article

0
/ / Recommend

Add to citation manager EndNote|Ris|BibTeX

URL: https://journal.psych.ac.cn/acps/EN/10.3724/SP.J.1041.2024.01634

https://journal.psych.ac.cn/acps/EN/Y2024/V56/I11/1634

Figures/Tables 15

Figure 1. Attribute hierarchy for the six attributes.

Figure 1. Attribute hierarchy for the six attributes.

Figure 2. PMR of three short tests encompassing six attributes and four attribute hierarchies that vary with s and g.

Figure 2. PMR of three short tests encompassing six attributes and four attribute hierarchies that vary with s and g.

Figure 3. Attribute hierarchy for the five attributes.

Figure 3. Attribute hierarchy for the five attributes.

Figure 4. Attribute hierarchy for the seven attributes.

Figure 4. Attribute hierarchy for the seven attributes.

Figure 5. Attribute hierarchy for the eight attributes.

Figure 5. Attribute hierarchy for the eight attributes.

Table 1 Three long tests with 6 attributes and s and g obeyed U (0,0.15)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9956	0.9736
	3(3)	0.9986	0.9919	0.9989	0.9932
	6(6)	0.9997	0.9981	0.9995	0.9972	0.9967	0.9803
Convergence structure (a)	1(2)	0.9952	0.9714	0.9953	0.9716
	2(4)	0.9993	0.9962	0.9964	0.9786
	3(6)	0.9987	0.9922	0.9987	0.9922	0.9958	0.9748
Convergence structure (b)	1(2)	0.9958	0.9747	0.9962	0.9774
	2(4)	0.9993	0.9960	0.9971	0.9829
	3(6)	0.9992	0.9952	0.9988	0.9930	0.9959	0.9752
Convergence structure ?	1(3)	0.9974	0.9844	0.9978	0.9865
Convergence structure ?	2(6)	0.9983	0.9898	0.9987	0.9923	0.9962	0.9780
Divergent structure	1(3)	0.9974	0.9848	0.9979	0.9876
Divergent structure	2(6)	0.9981	0.9884	0.9987	0.9922	0.9944	0.9665
Unstructure	1(5)	0.9964	0.9785	0.9980	0.9885
Unstructure	1⁺(6)	0.9977	0.9872	0.9986	0.9919	0.9932	0.9590

Table 1 Three long tests with 6 attributes and s and g obeyed U (0,0.15)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9956	0.9736
	3(3)	0.9986	0.9919	0.9989	0.9932
	6(6)	0.9997	0.9981	0.9995	0.9972	0.9967	0.9803
Convergence structure (a)	1(2)	0.9952	0.9714	0.9953	0.9716
	2(4)	0.9993	0.9962	0.9964	0.9786
	3(6)	0.9987	0.9922	0.9987	0.9922	0.9958	0.9748
Convergence structure (b)	1(2)	0.9958	0.9747	0.9962	0.9774
	2(4)	0.9993	0.9960	0.9971	0.9829
	3(6)	0.9992	0.9952	0.9988	0.9930	0.9959	0.9752
Convergence structure ?	1(3)	0.9974	0.9844	0.9978	0.9865
Convergence structure ?	2(6)	0.9983	0.9898	0.9987	0.9923	0.9962	0.9780
Divergent structure	1(3)	0.9974	0.9848	0.9979	0.9876
Divergent structure	2(6)	0.9981	0.9884	0.9987	0.9922	0.9944	0.9665
Unstructure	1(5)	0.9964	0.9785	0.9980	0.9885
Unstructure	1⁺(6)	0.9977	0.9872	0.9986	0.9919	0.9932	0.9590

Table 2 Three long tests with 6 attributes and s and g obeyed U (0,0.25)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9917	0.9539
	3(3)	0.9927	0.9577	0.9964	0.9785
	6(6)	0.9983	0.9898	0.9977	0.9865	0.9940	0.9638
Convergence structure (a)	1(2)	0.9904	0.9451	0.9917	0.9502
	2(4)	0.9970	0.9839	0.9933	0.9597
	3(6)	0.9964	0.9788	0.9960	0.9759	0.9928	0.9573
Convergence structure (b)	1(2)	0.9922	0.9545	0.9934	0.9605
	2(4)	0.9972	0.9846	0.9945	0.9674
	3(6)	0.9976	0.9856	0.9966	0.9796	0.9933	0.9600
Convergence structure (c)	1(3)	0.9948	0.9694	0.9956	0.9742
Convergence structure (c)	2(6)	0.9965	0.9793	0.9969	0.9818	0.9935	0.9614
Divergent structure	1(3)	0.9931	0.9610	0.9954	0.9734
Divergent structure	2(6)	0.9958	0.9751	0.9962	0.9780	0.9923	0.9543
Unstructure	1(5)	0.9937	0.9631	0.9958	0.9759
Unstructure	1⁺(6)	0.9953	0.9748	0.9959	0.9765	0.9912	0.9503

Table 2 Three long tests with 6 attributes and s and g obeyed U (0,0.25)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9917	0.9539
	3(3)	0.9927	0.9577	0.9964	0.9785
	6(6)	0.9983	0.9898	0.9977	0.9865	0.9940	0.9638
Convergence structure (a)	1(2)	0.9904	0.9451	0.9917	0.9502
	2(4)	0.9970	0.9839	0.9933	0.9597
	3(6)	0.9964	0.9788	0.9960	0.9759	0.9928	0.9573
Convergence structure (b)	1(2)	0.9922	0.9545	0.9934	0.9605
	2(4)	0.9972	0.9846	0.9945	0.9674
	3(6)	0.9976	0.9856	0.9966	0.9796	0.9933	0.9600
Convergence structure (c)	1(3)	0.9948	0.9694	0.9956	0.9742
Convergence structure (c)	2(6)	0.9965	0.9793	0.9969	0.9818	0.9935	0.9614
Divergent structure	1(3)	0.9931	0.9610	0.9954	0.9734
Divergent structure	2(6)	0.9958	0.9751	0.9962	0.9780	0.9923	0.9543
Unstructure	1(5)	0.9937	0.9631	0.9958	0.9759
Unstructure	1⁺(6)	0.9953	0.9748	0.9959	0.9765	0.9912	0.9503

Table 3 Three long tests with 6 attributes and s and g obeyed U (0,0.35)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9792	0.8901
	3(3)	0.9913	0.9506	0.9909	0.9460
	6(6)	0.9959	0.9761	0.9929	0.9581	0.9892	0.9358
Convergence structure (a)	1(2)	0.9795	0.8901	0.9864	0.9198
	2(4)	0.9896	0.9466	0.9882	0.9304
	3(6)	0.9903	0.9449	0.9906	0.9445	0.9884	0.9336
Convergence structure (b)	1(2)	0.9851	0.9170	0.9874	0.9287
	2(4)	0.9909	0.9521	0.9888	0.9364
	3(6)	0.9919	0.9538	0.9905	0.9465	0.9880	0.9322
Convergence structure (c)	1(3)	0.9872	0.9304	0.9892	0.9417
Convergence structure (c)	2(6)	0.9907	0.9490	0.9907	0.9500	0.9881	0.9327
Divergent structure	1(3)	0.9846	0.9188	0.9892	0.9402
Divergent structure	2(6)	0.9886	0.9341	0.9908	0.9479	0.9870	0.9256
Unstructure	1(5)	0.9859	0.9256	0.9895	0.9473
Unstructure	1⁺(6)	0.9861	0.9268	0.9894	0.9473	0.9854	0.9227

Table 3 Three long tests with 6 attributes and s and g obeyed U (0,0.35)

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9792	0.8901
	3(3)	0.9913	0.9506	0.9909	0.9460
	6(6)	0.9959	0.9761	0.9929	0.9581	0.9892	0.9358
Convergence structure (a)	1(2)	0.9795	0.8901	0.9864	0.9198
	2(4)	0.9896	0.9466	0.9882	0.9304
	3(6)	0.9903	0.9449	0.9906	0.9445	0.9884	0.9336
Convergence structure (b)	1(2)	0.9851	0.9170	0.9874	0.9287
	2(4)	0.9909	0.9521	0.9888	0.9364
	3(6)	0.9919	0.9538	0.9905	0.9465	0.9880	0.9322
Convergence structure (c)	1(3)	0.9872	0.9304	0.9892	0.9417
Convergence structure (c)	2(6)	0.9907	0.9490	0.9907	0.9500	0.9881	0.9327
Divergent structure	1(3)	0.9846	0.9188	0.9892	0.9402
Divergent structure	2(6)	0.9886	0.9341	0.9908	0.9479	0.9870	0.9256
Unstructure	1(5)	0.9859	0.9256	0.9895	0.9473
Unstructure	1⁺(6)	0.9861	0.9268	0.9894	0.9473	0.9854	0.9227

Table 4 The accuracy rates of three long tests with five attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9873	0.9229
	3(3)	0.9927	0.9533	0.9901	0.9506
	5(5)	0.9962	0.9696	0.9949	0.9685	0.9846	0.9244
Convergence structure	1(3)	0.9865	0.9358	0.9867	0.9377
Convergence structure	1⁺(5)	0.9924	0.9647	0.9896	0.9511	0.9915	0.9585
Divergent structure	1(2)	0.9812	0.9090	0.9856	0.9300
	2(4)	0.9906	0.9537	0.9873	0.9381
	2⁺(5)	0.9915	0.9591	0.9900	0.9518	0.9872	0.9375
Unstructure	1(4)	0.9856	0.9302	0.9881	0.9450
Unstructure	1⁺(5)	0.9870	0.9412	0.9890	0.9495	0.9859	0.9337

Table 4 The accuracy rates of three long tests with five attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9873	0.9229
	3(3)	0.9927	0.9533	0.9901	0.9506
	5(5)	0.9962	0.9696	0.9949	0.9685	0.9846	0.9244
Convergence structure	1(3)	0.9865	0.9358	0.9867	0.9377
Convergence structure	1⁺(5)	0.9924	0.9647	0.9896	0.9511	0.9915	0.9585
Divergent structure	1(2)	0.9812	0.9090	0.9856	0.9300
	2(4)	0.9906	0.9537	0.9873	0.9381
	2⁺(5)	0.9915	0.9591	0.9900	0.9518	0.9872	0.9375
Unstructure	1(4)	0.9856	0.9302	0.9881	0.9450
Unstructure	1⁺(5)	0.9870	0.9412	0.9890	0.9495	0.9859	0.9337

Table 5 The accuracy rates of three long tests with seven attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9853	0.9077
	4(4)	0.9899	0.9468	0.9892	0.9262
	7(7)	0.9952	0.9667	0.9929	0.9520	0.9905	0.9363
Convergence structure	1(2)	0.9796	0.8854	0.9883	0.9198
	2(4)	0.9914	0.9470	0.9898	0.9301
	3⁺(7)	0.9922	0.9515	0.9922	0.9463	0.9901	0.9337
Divergent structure	1(4)	0.9860	0.9097	0.9903	0.9377
Divergent structure	1⁺(7)	0.9910	0.9427	0.9906	0.9393	0.9877	0.9214
Unstructure	1(6)	0.9840	0.9143	0.9871	0.9273
Unstructure	1⁺(7)	0.9873	0.9271	0.9867	0.9264	0.9849	0.9058

Table 5 The accuracy rates of three long tests with seven attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9853	0.9077
	4(4)	0.9899	0.9468	0.9892	0.9262
	7(7)	0.9952	0.9667	0.9929	0.9520	0.9905	0.9363
Convergence structure	1(2)	0.9796	0.8854	0.9883	0.9198
	2(4)	0.9914	0.9470	0.9898	0.9301
	3⁺(7)	0.9922	0.9515	0.9922	0.9463	0.9901	0.9337
Divergent structure	1(4)	0.9860	0.9097	0.9903	0.9377
Divergent structure	1⁺(7)	0.9910	0.9427	0.9906	0.9393	0.9877	0.9214
Unstructure	1(6)	0.9840	0.9143	0.9871	0.9273
Unstructure	1⁺(7)	0.9873	0.9271	0.9867	0.9264	0.9849	0.9058

Table 6 The accuracy rates of three long tests with eight attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9841	0.9041
	4(4)	0.9906	0.9332	0.9934	0.9490
	8(8)	0.9925	0.9627	0.9937	0.9598	0.9927	0.9468
Convergence structure	1(3)	0.9852	0.8962	0.9918	0.9393
	2(6)	0.9899	0.9269	0.9914	0.9364
	2⁺(8)	0.9913	0.9369	0.9928	0.9485	0.9914	0.9347
Divergent structure	1(5)	0.9827	0.8864	0.9891	0.9256
Divergent structure	1⁺(8)	0.9883	0.9234	0.9893	0.9277	0.9899	0.9270
Unstructure	1(7)	0.9781	0.8499	0.9851	0.9089
Unstructure	1⁺(8)	0.9827	0.8957	0.9782	0.8973	0.9853	0.8939

Table 6 The accuracy rates of three long tests with eight attributes

Attribute hierarchy	Multiples with the simplest complete Q columns (total number of columns)	Contains one or more SSCQMS		Contains one or more USCQM		The reachability matrix
Attribute hierarchy		MMR	PMR	MMR	PMR	MMR	PMR
Linear structure	1(1)	0.9841	0.9041
	4(4)	0.9906	0.9332	0.9934	0.9490
	8(8)	0.9925	0.9627	0.9937	0.9598	0.9927	0.9468
Convergence structure	1(3)	0.9852	0.8962	0.9918	0.9393
	2(6)	0.9899	0.9269	0.9914	0.9364
	2⁺(8)	0.9913	0.9369	0.9928	0.9485	0.9914	0.9347
Divergent structure	1(5)	0.9827	0.8864	0.9891	0.9256
Divergent structure	1⁺(8)	0.9883	0.9234	0.9893	0.9277	0.9899	0.9270
Unstructure	1(7)	0.9781	0.8499	0.9851	0.9089
Unstructure	1⁺(8)	0.9827	0.8957	0.9782	0.8973	0.9853	0.8939

Figure 6. PMR of four attribute hierarchies and three short tests varying with the number of attributes.

Figure 6. PMR of four attribute hierarchies and three short tests varying with the number of attributes.

Figure 7. Hierarchical relationship of cognitive attributes of primary school travel problems.

Figure 7. Hierarchical relationship of cognitive attributes of primary school travel problems.

Table 7 Test items and their attributes

Test items	Attributes
Test items	A1	A2	A3	A4	A5	A6	A7	A8
1	1	0	0	0	0	0	0	0
2	0	1	0	0	0	0	0	0
3	1	0	1	0	0	0	0	0
4	0	1	0	0	0	0	1	0
5	1	1	0	0	0	0	0	0
6	1	1	1	0	1	0	0	0
7	0	1	0	0	1	0	0	0
8	0	1	0	0	0	1	0	0
9	0	1	0	0	1	1	1	0
10	1	1	1	1	0	1	0	1
11	1	1	1	1	0	1	1	0
12	1	1	1	1	1	1	1	0
13	1	1	1	1	0	0	0	0
14	1	1	1	1	0	0	1	0
15	1	1	1	1	1	1	1	1
16	1	1	0	0	1	0	0	0
17	1	1	1	1	1	0	1	0

Table 7 Test items and their attributes

Test items	Attributes
Test items	A1	A2	A3	A4	A5	A6	A7	A8
1	1	0	0	0	0	0	0	0
2	0	1	0	0	0	0	0	0
3	1	0	1	0	0	0	0	0
4	0	1	0	0	0	0	1	0
5	1	1	0	0	0	0	0	0
6	1	1	1	0	1	0	0	0
7	0	1	0	0	1	0	0	0
8	0	1	0	0	0	1	0	0
9	0	1	0	0	1	1	1	0
10	1	1	1	1	0	1	0	1
11	1	1	1	1	0	1	1	0
12	1	1	1	1	1	1	1	0
13	1	1	1	1	0	0	0	0
14	1	1	1	1	0	0	1	0
15	1	1	1	1	1	1	1	1
16	1	1	0	0	1	0	0	0
17	1	1	1	1	1	0	1	0

Table 8 The difference of participants’ mastery attributes estimated by Q1-Q3 and R (%)

Tests	Attributes
Tests	A1	A2	A3	A4	A5	A6	A7	A8
Q1	-5.0	-4.5	-0.48	0.56	-4.2	0.65	1.3	-0.24
Q2	-6.5	-3.6	-5.6	-2.2	-2.4	-1.8	-4.5	-2.3
Q3	1.3	1.4	1.5	2.2	1.5	3.3	2.1	-0.24

Table 8 The difference of participants’ mastery attributes estimated by Q1-Q3 and R (%)

Tests	Attributes
Tests	A1	A2	A3	A4	A5	A6	A7	A8
Q1	-5.0	-4.5	-0.48	0.56	-4.2	0.65	1.3	-0.24
Q2	-6.5	-3.6	-5.6	-2.2	-2.4	-1.8	-4.5	-2.3
Q3	1.3	1.4	1.5	2.2	1.5	3.3	2.1	-0.24

References

Metrics

Viewed

Full text

Abstract

Comments

Abstract
Figures/Tables
References
Related Articles
Recommended Articles
Metrics
Comments

TOP